Apparatus and method for determining co-location of services using a device that generates an audio signal

ABSTRACT

An audio signature is generated by a user device by generating an audio signal indicative of the user device, outputting the audio signal indicative of the user device through a speaker of the user device, using a microphone of the user device to sense audio, generating an audio signature of the audio sensed by the microphone in the user device, and transmitting the audio signature from the user device to a communication channel. The audio signal may be used by another user device which generates its own audio signature, and which can be compared to the audio signature of the user device that originally generated the audio signal to determine if the user devices are co-located.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of copending U.S. patent applicationSer. No. 14/713,928 filed May 5, 2015, the entire contents of which areincorporated by reference.

This application claims priority under 35 U.S.C. §119(e) to U.S.Provisional Application Ser. Nos. 61/994,799, 61/994,634, 61/994,805,61/994,810 and 61/994,812, all filed on May 16, 2014, the entirecontents of which are incorporated by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to a method of determining co-location of devicesusing audio information, and particularly a method which relies onidentification of unique or matching audio signatures to identifyco-located devices.

2. Description of the Related Technology

EP 2 501 145 A2 entitled, “Content Provision,” is expressly incorporatedby reference herein and shows an application where it is useful toidentify the audio or video content being presented to a user.

US20130205318 A1 entitled, “Method and system for automatic contentrecognition based on customized user preferences,” is expresslyincorporated by reference herein and shows an automatic contentrecognition (ACR)-enabled connected TV device may be operable toidentify, utilizing an ACR system, content in a presented program.

US20130071090 A1 entitled, “Automatic content recognition system andmethod for providing supplementary content,” is expressly incorporatedby reference herein and shows automatic content recognition performed todetermine the primary digital content. The time of a sample may also bedetermined. Supplementary digital content is then selected andtransmitted to the media device, or to another device, based upon theidentified primary digital content. The supplementary digital contentmay be adapted in layout, type, length, or other manners, based upon theplatform and/or configuration of the media device or any other device towhich the supplementary digital content is transmitted.

US20110289114 A1 entitled “System and method for auto contentrecognition,” is expressly incorporated by reference herein and showsautomatically recognizing media contents by the steps of capturing mediacontent from the Internet and/or devices, extracting fingerprints fromcaptured content and transferring to backend servers for identification,and backend servers processing the fingerprints and replying withidentified result.

SUMMARY OF THE INVENTION

According to an advantageous feature of the invention an audio signaturemay be used to characterize the audio at a location for automaticcontent recognition.

An Audio Signature is a digital summary, generated from an audio signal,which can be used to identify an audio sample or quickly locate similaritems in a database.

An Audio Fingerprint is a condensed digital summary, generated from anaudio signal, which can be used to identify an audio sample or quicklylocate similar items in an audio database. As used herein, a fingerprintis a special form of a signature.

Automatic content recognition (ACR) refers to the ability to identify acontent element within the proximity of a probe or sensor, audio, videoor image, based on sampling a portion of the audio, or video, or image,processing the sample and comparing it with a reference.

User devices include audio detection components, such as microphones,may be portable. It is an object of the invention to determine when suchdevices are co-located.

It is an object of the invention to facilitate determining co-locationof user devices.

Co-location means sufficiently close in proximity so as to be consideredto be together, in light of the purpose of an inquiry. Variations insampling devices and acoustical differences even in very close togetherlocations will cause differences in audio signatures generated bydifferent devices. The process of comparison of audio signatures is notintended to identify an exact match. Digital representations ofsignatures which match within a tolerance may be considered to beco-located. An empirical study may be used to determine the tolerance byexperimental readings under various conditions, distances and deviceshow much tolerance will be permitted in the comparisons to justify aninference of co-location.

An apparatus for determining co-location of devices may include areceiver connected to one or more communication channels configured orprogrammed to receive a remote audio signature and a remote deviceidentification over a communication channel and a database. One or moreremote audio signatures may be stored in the database. A remote deviceidentification may be associated with an audio signature stored in thedatabase. A query engine may be connected to the database configured orprogrammed to compare a remote audio signature stored in the database toa remote audio signature received over the communication channel Theapparatus may include a processor responsive to the query engineconfigured or programmed to report a remote device identificationassociated with one of the audio signatures when an audio signaturereceived over the communication channel matches an audio signaturestored in the database within a threshold confidence level. Theprocessor may be configured or programmed to report a remote deviceidentification associated with an audio signature received over thecommunication channel and a remote device identification associated withan audio signature stored in the database when the audio signaturesmatch within a threshold confidence level.

A component of the audio signature may represent a remote deviceidentification. The query engine may be configured or programmed tolimit a query set of the database according to a specified parameter.The specified parameter may be a temporal parameter. The specifiedparameter may be based on location services. The specified parameter mayrequire temporal and location services based proximity of the audiosignature received over the communication channel and the audiosignature stored in the database.

A database controller may be configured or programmed to remove expiredaudio signatures from a comparison set. A database controller may beconfigured or programmed to limit matching queries to no more than apreset number of stored audio signatures. A database controller may beconfigured or programmed to delete audio signatures in the databaseafter a predetermined time period.

A method for determining co-location of devices may include the steps ofreceiving a remote audio signature and remote device identification overthe communication channel, comparing a remote audio signature stored inthe database to a remote audio signature received over the communicationchannel, and reporting a remote device identification associated withone of the audio signatures when an audio signature received over thecommunication channel matches an audio signature stored in the databasewithin a threshold confidence level.

The method may include the step of reporting a remote deviceidentification associated with an audio signature received over thecommunication channel and a remote device identification associated withan audio signature stored in the database when the audio signaturesmatch within a threshold confidence level. A component of the audiosignature may represent a remote device identification. The method mayinclude the step of limiting a query set of the database according to aspecified parameter. The specified parameter may be a temporalparameter. The specified parameter may be based on location services.The specified parameter may require temporal and location services basedproximity of the audio signature received over the communication channeland the audio signature stored in the database. The method may includethe step of removing expired audio signatures from a comparison set. Themethod may include the step of limiting matching queries to no more thana preset number of stored audio signatures. The method may include thestep of deleting audio signatures in the database after a predeterminedtime period.

An apparatus for generating an audio signature may include a processor,a microphone connected to the processor, an audio signal generatorconnected to the processor and configured or programmed to generate anaudio signal indicative of the apparatus, a speaker responsive to theaudio signal generator, an audio signature generator connected to theprocessor and responsive to the microphone, and a transmitter connectedto the processor capable of transmitting an audio signature to acommunications channel.

The audio signal generator may be configured or programmed to generatean inaudible audio signal. The audio signal generator may be an audioidentification signal generator.

A method for generating an audio signature may include the steps ofgenerating an audio signal indicative of a user device, outputting theaudio signal indicative of a user device through a speaker, using amicrophone to sense audio, generating an audio signature of the audiosensed by the microphone, and transmitting the audio signature to acommunication channel. The step of generating may generate an inaudibleaudio signal. The step of generating may generate an inaudible audiosignal which may be an audio identification signal.

The invention may include an article of manufacture, a method, a system,and an apparatus for determining co-location of user devices.

The article of manufacture of the invention may include acomputer-readable medium comprising software for a system fordetermining co-location of user devices, using automatic contentrecognition.

The article of manufacture of the invention may include acomputer-readable medium comprising software for an automatic contentrecognition system, comprising code segments for generating audiosignatures and storing them with time stamps.

The article of manufacture of the invention may include acomputer-readable medium comprising software for an automatic contentrecognition system, comprising code segments for comparing generatedaudio signatures to reference signatures for identification anddetermining co-location of user devices.

The system of the invention may include a computer system including acomputer-readable medium having software to operate a computer or otherdevice in accordance with the invention.

The apparatus of the invention may include a computer including acomputer-readable medium having software to operate the computer inaccordance with the invention.

The article of manufacture of the invention may include acomputer-readable medium having software to operate a computer inaccordance with the invention.

Various objects, features, aspects, and advantages of the presentinvention will become more apparent from the following detaileddescription of preferred embodiments of the invention, along with theaccompanying drawings in which like numerals represent like components.

Moreover, the above objects and advantages of the invention areillustrative, and not exhaustive, of those that can be achieved by theinvention. Thus, these and other objects and advantages of the inventionwill be apparent from the description herein, both as embodied hereinand as modified in view of any variations which will be apparent tothose skilled in the art.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic of a system for automatic content recognition;

FIG. 2 schematically shows a wide geographic area having many portabledevices such as cell phones.

FIG. 3 shows a schematic of an embodiment of a system utilizingautomatic content recognition to determine co-location of user devices;

FIG. 4 shows a schematic of a remote user device having an audiogenerating feature; and

FIG. 5A shows a process flow diagram of an embodiment for determiningco-location of remote user devices; and

FIG. 5B shows a schematic process flow of audio signature generation andtransmission by a remote user device.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Before the present invention is described in further detail, it is to beunderstood that the invention is not limited to the particularembodiments described, as such may, of course, vary. It is also to beunderstood that the terminology used herein is for the purpose ofdescribing particular embodiments only, and is not intended to belimiting, since the scope of the present invention will be limited onlyby the appended claims.

Where a range of values is provided, it is understood that eachintervening value, to the tenth of the unit of the lower limit unlessthe context clearly dictates otherwise, between the upper and lowerlimit of that range and any other stated or intervening value in thatstated range is encompassed within the invention. The upper and lowerlimits of these smaller ranges may independently be included in thesmaller ranges is also encompassed within the invention, subject to anyspecifically excluded limit in the stated range. Where the stated rangeincludes one or both of the limits, ranges excluding either or both ofthose included limits are also included in the invention.

Unless defined otherwise, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. Although any methods andmaterials similar or equivalent to those described herein can also beused in the practice or testing of the present invention, a limitednumber of the exemplary methods and materials are described herein.

It must be noted that as used herein and in the appended claims, thesingular forms “a”, “an”, and “the” include plural referents unless thecontext clearly dictates otherwise.

All publications mentioned herein are incorporated herein by referenceto disclose and describe the methods and/or materials in connection withwhich the publications are cited. The publications discussed herein areprovided solely for their disclosure prior to the filing date of thepresent application. Nothing herein is to be construed as an admissionthat the present invention is not entitled to antedate such publicationby virtue of prior invention. Further, the dates of publication providedmay be different from the actual publication dates, which may need to beindependently confirmed.

The system may rely on automatic content recognition technology. Oneexample of the use of automatic content recognition technology is theidentification of audio. Automatic content recognition technology can beused to identify cable or network broadcast content. The shows mayinclude portions which are the episodes, referred to as principlecontent, and supplemental content such as commercials, emergency alertsand/or news interruptions. The content may be audio and visual, visualonly, or audio only. The embodiments of the invention are described interms of using audio as a representation of audio visual content, butthe invention is equally applicable to audio only content. Furthermorethe embodiments of the invention are described in the context ofsampling audio and generating audio signatures or fingerprints, but isequally applicable to sampling visual content, including video orimages, and generating signatures or fingerprints based on the visualsamples.

According to an advantageous feature of the invention, automatic contentrecognition may be utilized to compare audio sampled by one remote userdevice to audio sampled by another remote user device. Comparisons thatmatch with a sufficient level of confidence may permit a conclusion thatthe remote user devices are co-located.

FIG. 1 shows a schematic of a system for automatic content recognition.FIG. 1 shows a remote user device 180 which may be associated with anindividual or a location. The remote user device 180 may be a deviceused to collect or process audio information. The remote user device 180may be a smart phone, tablet or other device.

FIG. 1 shows a remote user device 180 having a probe 100 which may beused to collect information for use in automatic content recognition.The information may include audio information and other information thatis metadata related to the collected audio or associated with the remoteuser device or user. The probe may comprise more than one componentdepending on what components would be useful to collect the information.The probe may include a microphone, clock or time stamp generator,camera, GPS sensor or other elements for location services ortemperature sensor and may have access to other device information suchas user identification, serial number and MAC address, and time.

According to a preferred embodiment, the probe may be embodied in asmartphone or tablet or a smart glass (like google glass) or a smartwatch or a wearable device which may be associated with a user. Otherdevices embodying a probe may include, notebooks, desktop computers,dedicated devices or set-top boxes which have a microphone and/or camerato collect audio and/or video information. The probe may utilize aninternal clock or a network accessible clock to generate a time stamp.Smartphone devices provide ready access to location services and otherparameters which are known or derived by the probe device or accessedover a communications channel. Apple Support Note HT5594, the content ofwhich is incorporated herein, accessible athttp://support.apple.com/kbHT5594, describes location services availableto an iOS device.

The information may be processed in order to format and segregate theinformation. Formatted data representative of an audio or videopresentation may be processed by pre-processor 110. The pre-processoradvantageously is running on the same device as the probe and may be inthe form of an app on a smartphone or tablet. The pre-processor may be acomponent of an app running on a user's smartphone or tablet. Forexample, the pre-processor may control audio sampling to collect digitalinformation representative of audio.

The remote user device may capture audio information using a componentto “listen” to audio being played or present in the location of theremote user device 180. According to an advantageous feature, capturingmay be accomplished by audio sampling. The audio samples may betransmitted to a server as captured, or a signature may be generatedwhich may then be transmitted to a server location. The signature may bean audio fingerprint and may include metadata.

The remote user device need not be in the same location as a server. Theremote device 180 may be a cellphone or a tablet device or a wearabledevice. Smartphones such as an iPhone or Android-based device may serveas the remote user device 180. Some remote user devices may be stationedat a location. The stationed remote user devices may be implemented inor combined with a set-top box, media server, and home computer orDVD/Blu-ray player.

The server may operate to correlate the audio information received froma remote user device and reference information stored in a database toidentify or characterize the audio information. In particular, an audiosignature received from a remote user device may be compared tosignatures previously stored in a database. A signature match within anacceptable tolerance may indicate co-location of remote user devicescorresponding to matched audio signatures. The audio signatures from theremote user device may be stored in the database to compare to audiosignatures received from other remote user devices. The reference mayinclude audio signatures or audio fingerprints and metadata for one ormore streams of audio and/or visual programming, and may includeidentification of content and content class, such as principle contentor supplemental content.

This information may be useful for many purposes including, withoutlimitation, to associate devices in a household, to feed supplementalmedia to a user or user device, to price audio content delivery, foranalytic or other purposes. A correlation may be used to infer that twoor more remote audio devices are co-located.

FIG. 1 shows a diagram of an automatic content recognition (“ACR”)system. Automatic content recognition can be used in many applicationswhere it is desirable to identify audio information.

The pre-processor may pass the formatted audio samples to an audiosignature generation unit. There are a number of ways to generate anaudio signature. For example, U.S. Pat. No. 8,468,357 entitled,“Multiple Step Identification of Recordings” is expressly incorporatedherein by reference. U.S. Pat. No. 8,468,357 discloses a method andsystem where algorithmically-determined fingerprints are extracted fromaudio information utilizing one or more fingerprint extraction methods.An audio fingerprint may be used as an audio signature. The fingerprintmay be less unique than an uncompressed signature, but has the benefitof conserving transmission bandwidth and computational resources usedfor comparison to a reference data-set.

U.S. Pat. No. 8,453,170 B2 entitled, “System and method for monitoringand recognizing broadcast data,” is expressly incorporated by referenceherein and shows an automatic content recognition (ACR) system.

The audio signature generator 120 may be advantageously executed on thesame device as the probe 100 and the pre-processor 110. Advantageouslythat device is a user's cellphone, smartphone, or tablet. The audiosignature created by audio signature generator 120 and the metadatasegregated by pre-processor 110, if any, may be combined and transmittedby transmitter 101 to a server 170. A time stamp may be associated withthe audio signature by the remote user device 180. Alternatively, atimes stamp may be associated with an audio signature by the server 170upon receipt of the signature.

The server may include a receiver 102 receive an audio signature and adatabase controller 130. The receiver receives audio signatures and anymetadata associated with the audio signatures that is transmitted by aremote user device 180. The receiver 102 may associate a time stamp withan audio signature. The database controller 130 may advantageouslyformat the audio signature and associated metadata, if any, for storagein a database 140. The database controller 130 may also inform a queryengine 150 of the signature or the signature received from a user device180 (“query audio signature”) and metadata. The query engine 150 canadvantageously access the database 140 in order to find referencesignature(s) corresponding to the query audio signature from a remoteuser device 180 which resulted in the signature. When an audio signatureis matched, some or all of the metadata and or audio signature may bepassed to a result processing module 160 which is able to make use ofthe identification of a match and the metadata associated with the queryaudio signature and/or associated with the reference audio signature.

The database 140 may contain reference audio signatures. The audiosignature received from a remote user device 180 may be used to querythe database 140. The reference audio signatures may be audiofingerprints. The metadata may include information identifying theorigin of the audio signature, information identifying the contentcorresponding to the audio signature, distribution time and locations,and/or any other useful metadata.

EP 2 501 145 A2, the disclosure of which is incorporated herein, showsthat the identification of the content may be utilized to servesupplemental content to a user. The result processing system 160 may,for example, be a content augmentation service.

The remote user device 180, as previously discussed, may be utilized tomonitor audio information at a user location. A device 180 may also beprovided to monitor reference content and generate reference audiofingerprints and audio signatures with metadata which may be passed tothe database controller 130 and stored in database 140 as a reference.The metadata may be a time stamp, an indication of the identification ofthe reference content or an indication of the device that captures thecontent. If the audio signature derived at a user device 180 matches areference audio signature stored in the database, with an acceptablelevel of confidence, then the server 170 may use the metadata associatedwith the reference audio signature and/or associated with the captureddevice audio signature or derive the identification of the media contentmonitored by the remote user device 180.

The invention relates to a method and system for determining co-locationof remote user devices 180. Remote user devices may be stationary orportable. Portable devices may include cellular telephones, tablets,laptops or other devices having computational abilities and including amicrophone. Stationary devices may include computers, set top Boxes,dedicated signature generators and other devices. Often the location ofa portable electronic device is indicative of the location of its owneror user. It may be useful to ascertain when more than one device isco-located and it may be useful to ascertain the identity of co-locateddevices, and by inference, co-located users. A stationary device maygenerate audio signatures at the location of the stationary device. If aportable remote user device audio signature matches the audio signatureof the stationary device, the location of the portable remote userdevice may be inferred.

According to the invention, a portable electronic device may captureaudio information which is indicative of a localized position of adevice. That information may be transmitted to a server which comparesthe audio information with audio information received from other devicesin order to determine if two or more devices are co-located.

The method, according to the invention, may determine co-location withina relatively small area based on the similarity of captured audioinformation. This can isolate devices to a much smaller area thanlocation-based services such as GPS, cellular or Wi-Fi monitoring (i.e.through SSID), IP address based tracking or other indoor locationtracking systems.

There are several different modes of operation. According to one mode ofoperation. A remote user device 180 may emit a unique or pseudo-uniqueaudio signal. That device and other co-located devices may then capturethe audio signal and the information representative of the capturedaudio signals may be transmitted to a server which compares availableaudio signals to determine co-location. According to an advantageousembodiment, the portable devices may utilize the captured audio togenerate the audio signature which may be transmitted with or instead ofthe captured audio signal. The use of audio signatures allows acomputationally more efficient comparison of information indicative ofcaptured audio. The signature may be created by the portable device andtransmitted, or may be created at a server on the basis of transmittedaudio samples. According to an advantageous feature of the invention,the unique or pseudo-unique audio ID may be outside the audible rangewhich can be normally heard by people. The device ID may be assigned tothe individual device or generated on the basis of devicecharacteristics. The device characteristics may be a signature devicecomponent's MAC address or other information associated with aparticular device.

According to another embodiment of the invention, audio informationwhich is sufficiently detailed to identify a particular location basedon sound present in that location may be used to match audio informationreceived from two or more portable devices. The use of audio signaturesgenerated from audio information may facilitate the establishment ofsufficiently unique location's specific identifying information. Thismay be accomplished through monitoring the ambient sound available to aparticular portable device or one or more of the devices may generate asound to facilitate identification of signals from two or more deviceswhich are co-located. Because of slight variations in location,processing power, device characteristics and transmissioncharacteristics, the information indicative of the audio available to aportable device may not have the same precise arrival time at a systemfor performing comparison of information indicative of available audioinformation. According to an advantageous embodiment, signals which aresufficiently similar and sufficiently close in time may be considered tobe a match while still allowing for small variations in signalinformation and time.

According to an additional advantageous feature, location serviceinformation may be utilized to narrow the information subject tocomparisons. Location information services generally can identify abroader geographic area than is necessary to determine co-location, suchas devices in the same room. Location services cannot alone be utilizedto identify whether devices in the same vicinity are in fact co-located.The location service information can however be utilized to limit therequired comparisons and reduce the level of uniqueness required toestablish a match. Location services may be based on GPS, Wi-Fi and/orcellular facilities, or indoor location tracking services.

FIG. 2 schematically shows a wide geographic area having many portabledevices such as cell phones. Remote user devices A-G are shown asexamples. Remote user devices A and B may be co-located in a room whichmay have a television. Remote user device C may be located across townin a house with a television playing the same content at the same time.Remote user devices D, E & F may be located in vehicles traveling alongthe interstate and remote user device G may be in an airplane. Each ofthe devices may capture the available audio information by samplingavailable audio and according to a preferred embodiment may generate anaudio signature indicative of the sampled audio information. The audiosignatures may each be transmitted to a server. The server may ascertainif the signatures of two or more devices are sufficiently similar to beindicative of co-location. It is the intent to be able to identify thatdevices A and B are located and distinguished from all other devicestransmitting information.

Different ambient audio information may be available at each location.For example, Remote user device H is relatively close to Remote userdevices A and B, but Remote user device H may not detect sufficientlysimilar audio information to be considered a match.

Remote user device C may be located in a location with the same dominantaudio source as is present in the location of Remote user devices A andB. There may be other audio contributors to Remote user device C thatare not making the same contribution to the audio detected by Remoteuser devices A and B. Different audio contributors will result indifferent audio signatures and may be sufficient to distinguish theaudio detected in the respective locations. The same is true for Remoteuser devices D, E, F and G which are all subjected to the audioinformation present in their respective locations.

In order to reduce the complexity and number of audio signaturecomparisons which the server must make, it is possible to utilizelocation services. The location services may be useful to narrow therange of comparisons.

For example, available location services may be sufficiently broad toidentify Remote user devices A, B and H as being the same location onthe basis of the resolution of the location services. Similarly, Remoteuser devices D and E may be in the same location based on the resolutionof the available location services.

In this simplified embodiment, comparison of audio signatures from 8devices without any location-based service narrowing would require 28comparisons (A-B; A-C; A-D; A-E; A-F; A-G; A-H; B-C; B-D; B-E; B-F; B-G;B-H; C-D; C-E; C-F; C-G; C-H; D-E; D-F; D-G; D-H; E-F; E-G; E-H; F-G;F-H; and G-H.) In this simplified example, utilizing locations servicesnarrows the number of comparisons to 4 (A-B; A-H; B-H; and D-E.) Thisyields a very large computational savings may be achieved throughlocation based narrowing.

The apparatus and method may be used to determine when two user devicesare located within the same immediate vicinity. The system and methodmay be utilized to determine when two users are located in the same roomor within a similarly close proximity. Co-location of users is inferredwhen their remote user devices such as cellphones or tablets are exposedto the same ambient audio information at the same time.

FIG. 3 shows a system according to an embodiment of the invention.Remote user devices 280 and 281 may be devices associated withparticular individual users and may be smartphones or tablets. Only twodevices are shown, however it is understood that the system may receiveinformation from many user devices. The remote user devices may eachinclude a probe 200, 201 utilized to collect information. The probe mayinclude a microphone in order to sample of audio. The probe may alsoaccess a clock to collect a timestamp corresponding to an audio sampleand may access location services such as GPS, Wi-Fi and/or local networkinformation or cellular location services. The timestamp and locationservice information is metadata for the audio sample. Other metadataaccessible to the probe may also be collected.

The available location services information is generally notsufficiently localized in order to allow a conclusion that two or moreremote user devices are within the same immediate vicinity orco-located. The location services information and/or timestamp may beutilized to narrow the relevant data for investigation of co-location.

The probe may be connected to preprocessing units 210, 211 which preparethe captured metadata for transmission and the captured audio for audiosignature generation. The audio signatures may be fingerprints.

The preprocessing modules 210, 211 may pass the audio samples to anaudio signature generation unit 220, 221 and pass the metadata totransmitter 280, 281.

A server 270 may include a receiver 290. The receiver 290 may beconnected to a communications link and receive audio signaturesgenerated at a remote user device over the communication link. In thesame manner as shown in FIG. 1, the audio signatures are passed to adatabase controller 230 which may be responsible for storing andretrieving audio signatures from the database 240.

A query engine 250 may be provided to process the audio signaturesreceived from one or more remote user devices against audio signaturesstored in the database 240 along with the result processing unit 260.The system according to FIG. 3, may operate to store audio signaturesreceived from remote user devices into the database and to compare audiosignatures received from remote user devices 280, 281 to audiosignatures already in the database in order to identify audio signaturesfrom remote user devices which are close enough to indicate co-locationof the devices.

FIG. 4 shows a schematic of components in a remote user device 320 whichis particular suited for use in a system for determining co-location ofremote user devices. In addition to the structure described for remoteuser devices 280, 281, remote user device 320 may takes advantage of anaudio ID generation module 300. The audio ID generation module 300 maygenerate an identification signal which may be output to a speaker 310of the remote user device 320. The audio ID may be utilized with orinstead of the ambient audio being sampled by the probe. According toone embodiment, the audio ID may be a unique identification for theremote user device. The audio ID may be assigned to the remote userdevice or derived from characteristics, features or structure of theremote user device. For example, telephone number, MAC ID or User IDcould serve as the basis for an audio ID signal.

The audio ID need not be unique and instead may be sufficient to allowthe server to process signatures received from remote user devices toindicate presence of another device in the same location with asufficient level of confidence for the system. The audio ID need only beunique enough for the server to determine, with an acceptable level ofconfidence, that a remote user device is present in the same vicinity asanother remote user device. Advantageously the audio ID may be in alimited frequency range to ease the technical complexity of summarizingthe audio.

FIG. 5A shows a process that may be utilized to generate audiosignatures at a remote user device. A remote user device 180 mayestablish a schedule to sample audio at 301. The capture may be by anaudio sampling technique and the samples may be utilized to generate anaudio signature at step 302. In addition, metadata may be added to theaudio signature generated at step 302. The audio signature may then betransmitted to a server at step 410.

FIG. 5B shows a process for determining co-location according to oneembodiment of the invention. The process may have a start 400. Theserver is advantageously implemented in a general purpose computer butmay be accomplished using special purpose hardware. An audio signatureis received at step 411. The audio signature may be from a remote userdevice. The audio signature may be compared to signatures in thedatabase. When a received signature is sufficiently similar to asignature already in the database, the system concludes that the remoteuser device associated with the received signature and the remote userdevice associated with the stored signature are co-located. If thereceived signature is not sufficiently similar to a signature already inthe database, then the received signature may be added to the databasefor comparison with subsequently received signatures and associated withthe source remote user device.

According to an advantageous embodiment, the database set of signaturesutilized for comparison may be limited to signatures having timestampswhich are close enough in time to the received signature to lead to amatch. Alternatively, signatures that have a timestamp offset from thereceived signature timestamp that permit a conclusion that they do notmatch within an acceptable probability may be excluded from thecomparison set.

According to an alternative embodiment, signatures in the database mayexpire or be expunged when they are sufficiently stale to be unlikely tobe matched with any received signature with an acceptable level ofcertainty.

According to one embodiment, a location service inquiry 420 may be usedto narrow the comparison set. In this embodiment, the audio signaturesreceived and stored in the database may include metadata obtained fromlocation services indicative of the general location of a remote userdevice. There is no need to query the database for fingerprint matchingif the location services metadata does not indicate that the devices arewithin the same general location.

If the comparison set in the database does not include any signatureshaving the same location services information at 430, the audiosignature may be added to the database at 490 for inclusion insubsequent comparison sets. If the location services information of thereceived signature is determined to match at 430 with one or more audiosignatures in the comparison set, then the comparison set can benarrowed to include only the matching signatures.

Next, the timestamp of the received audio signature may be compared tothe timestamp of audio signatures in the comparison set at 440. Anysignatures in the comparison set that have timestamps outside of anacceptable range for the purpose of comparison may be excluded at 440.If the system, at 450, determines that there are no audio signatures inthe comparison set that are acceptably proximal in time to the timestampof the received audio signature, the received audio signature may beadded to the database at 490 for inclusion in subsequent comparisonsets.

An audio signature inquiry performed at step 460 may be morecomputationally-intensive than a location service query or a timestampquery. If there are no audio signatures in the reference setsufficiently similar to the received audio signatures on the basis of asignature comparison determined at 470, the received audio signature maybe included in the database for use in subsequent comparison sets. Ifthe fingerprint of a received audio signature is sufficiently close to afingerprint contained in the comparison set, the system may concludethat the remote user device associated with the received audio signatureis co-located with the remote user device associated with the matchedsignature in the comparison database.

Step 480 may report co-location of the respective devices. The reportmay then be utilized. After a report of co-location at step 480 or thestorage of an unmatched audio signature in the database at 490, theprocess may return to the start 400 to receive the next audio signature.

According to an embodiment of the invention, the received audiosignatures may be buffered or placed in temporary storage before beingprocessed. Alternatively, the received audio signatures may be saved tothe database and the system will keep track of which, or select audiosignatures for use as the received audio signature in the method shownin FIG. 4 and/or to select the stored signatures to be used in acomparison set.

According an embodiment of the invention, the received and stored audiosignatures may be prequalified by using only signatures which arederived from samples of specified audio content. For example, the systemmay be utilized to determine when two or more remote user devices areco-located during the play of a particular piece of media, for example,a commercial.)

The invention is described in detail with respect to preferredembodiments, and it will now be apparent from the foregoing to thoseskilled in the art that changes and modifications may be made withoutdeparting from the invention in its broader aspects, and the invention,therefore, as defined in the claims, is intended to cover all suchchanges and modifications that fall within the true spirit of theinvention.

Thus, specific apparatus for and methods of auto-populating smarttemplates with data from multiple sources with structured andunstructured data have been disclosed. It should be apparent, however,to those skilled in the art that many more modifications besides thosealready described are possible without departing from the inventiveconcepts herein. The inventive subject matter, therefore, is not to berestricted except in the spirit of the disclosure. Moreover, ininterpreting the disclosure, all terms should be interpreted in thebroadest possible manner consistent with the context. In particular, theterms “comprises” and “comprising” should be interpreted as referring toelements, components, or steps in a non-exclusive manner, indicatingthat the referenced elements, components, or steps may be present, orutilized, or combined with other elements, components, or steps that arenot expressly referenced.

What is claimed is:
 1. An apparatus for generating an audio signaturecomprising: a probe including a microphone; an audio signal generatorconfigured to generate an audio signal indicative of said apparatus; aspeaker responsive to said audio signal generator that outputs thegenerated audio signal; an audio signature generator connected to saidmicrophone configured to generate an audio signature from a sampledaudio signal detected by the microphone, wherein the audio signaturegenerator generates the audio signature without requiring detection ofany additional audio signal; and a transmitter connected to said audiosignature generator configured to transmit the audio signature to acommunications channel.
 2. An apparatus according to claim 1 whereinsaid audio signal generator is configured to generate an inaudible audiosignal.
 3. An apparatus according to claim 1 wherein said audio signalgenerator is an audio identification signal generator.
 4. An apparatusaccording to claim 1 wherein said audio signal is an inaudible audiosignal which is an audio identification signal.
 5. An apparatusaccording to claim 3 wherein the audio identification signal generatorgenerates a unique identification for the apparatus.
 6. An apparatusaccording to claim 1 wherein the apparatus is a smartphone or a tablet.7. An apparatus according to claim 1 wherein the probe provides atimestamp corresponding to the sampled audio signal, the timestamp beingtransmitted with the audio signature.
 8. An apparatus according to claim1 wherein the probe provides location data corresponding to the sampledaudio signal, the location data being transmitted with the audiosignature.
 9. A method for generating an audio signature by a userdevice comprising the steps of: generating an audio signal indicative ofthe user device; outputting said audio signal indicative of the userdevice through a speaker of the user device; using a microphone of theuser device to sense audio; generating an audio signature of the audiosensed by the microphone in the user device, wherein the audio signatureis generated without requiring detection of any additional audio signal;and transmitting said audio signature from the user device to acommunication channel.
 10. A method according to claim 9 wherein saidstep of generating an audio signal, generates an inaudible audio signal.11. A method according to claim 9 wherein the audio signal indicative ofthe user device is a unique identification of the user device.
 12. Amethod according to claim 9 wherein the user device is a smartphone or atablet.
 13. A method according to claim 9 further comprising the stepof: providing by the user device a timestamp corresponding to the sensedaudio signal, the timestamp being transmitted with the audio signature.14. A method according to claim 9 further comprising the step of:providing by the user device location data corresponding to the sensedaudio signal, the location data being transmitted with the audiosignature.
 15. A method for determining co-location of multiple devices,a first device including an audio signal generator and a speakerresponsive to the audio signal generator that outputs the generatedaudio signal, the first device and a second device each including aprobe having a microphone that receives the audio signal, an audiosignature generator responsive to the microphone, and a transmitterconnected to the audio signature generator, the method comprising: (a)the first device outputting the generated audio signal; (b) the firstand second device performing the steps of: (i) generating by the audiosignature generator of the respective device an audio signature from theaudio signal received by the microphone of the respective device,wherein the audio signature generator of the respective devicesgenerates the audio signature without requiring detection of anyadditional audio signal, and (ii) transmitting by the transmitter of therespective device the generated audio signature and deviceidentification of the respective device to a communication channel; (c)receiving at a remote server connected to the communication channel theaudio signatures of the respective devices and the device identificationof the respective devices; (d) comparing the audio signatures of therespective devices and determining whether the audio signatures matchone another within a threshold level of confidence; and (e) reportingthe device identification of devices having matching audio signatures,wherein the matching audio signatures identify likely co-locateddevices.
 16. The method of claim 15 wherein the audio signal generatorof the first device is configured to generate an inaudible audio signal,wherein steps (b)-(d) are performed using the audio signature of theinaudible audio signal.
 17. The method of claim 15 wherein the firstdevice is a smartphone or tablet.
 18. The method of claim 15 wherein thefirst and second devices are both either smartphones or tablets.
 19. Themethod of claim 15 wherein the probe of each device further provides atimestamp corresponding to the audio signal received by the microphoneof the respective device, the timestamp being transmitted with the audiosignature, the method further comprising: (f) comparing the timestampsfrom the respective devices, wherein the comparison step (d) isperformed only for audio signatures that have timestamps within apredefined proximity of time.
 20. The method of claim 15 wherein theprobe of each device further provides location data corresponding to theaudio signal received by the microphone of the respective device, thelocation data being transmitted with the audio signature, the methodfurther comprising: (f) comparing the location data from the respectivedevices, wherein the comparison step (d) is performed only for audiosignatures that have the same location data.
 21. The method of claim 20wherein the location data is based on one of GPS, Wi-Fi, cellularfacilities, or indoor location tracking services.
 22. The method ofclaim 15 wherein the generated audio signal does not uniquely identifythe first device.